The interactive word: reclaiming multimedia for text

The paper is concerned with visual and interaction design using text. It highlights the current neglect of text as an interface element, and suggests reasons why this may be so. It offers some accounts of the role of visual design for text and recommends one approach. A review is made of the strengths and weaknesses of current display typography in relation to designing interactive multimedia. Finally, the potential for using text as a fully integrated component of multimedia products is highlighted using two projects undertaken in the Centre for Electronic Arts at Middlesex University. Introduction The aim of this paper is to help reinstate text as a means of communication in multimedia, and to propose some ways in which its design might be better conceived. It is the author’s belief that there is a lack of serious attention to the uses of text in interactive multimedia, and that the poor design of such text needs to be questioned. Text neglected Practitioners and theorists of multimedia have become fascinated by all media except text. A sample survey of the proceedings of three human-factors conferences reveals no article dedicated to text in human computer interaction (INTERCHI 1993, SIGCHI ’94, ’95); nevertheless there is still a widespread, unspoken assumption that text is essential. Text is mentioned as part of just two comparative studies, which while they emphasise the value of graphics in assisting communication, acknowledge that text cannot be replaced (Ogozalek 1994, Harrison 1995). Turning to some standard text books we find that Laurel, Oren and Don (1992) mention text in the context of media integration, but significantly only to imply that making decisions about text is easier than making decisions about other media such as video. When text is discussed, either its appearance is ignored, or the discussion harks back to simple guidelines about text on paper (Loo 1991, Henno 1993, Hanneman and Thuring 1995, Hardman 1995). Certainly, the potential for using text as an integrated component of the interactive interface is ignored. No account is taken of improvements in text-display technology; current thinking about the nature of interfaces; or the value of visual rhetoric in articulating the interface. We hope to address some of these issues in this paper. Why has text become the Cinderella of multimedia? A number of explanations are possible. • Text on screens is not a novelty, being the oldest medium used on computer displays: even the numbers displayed by early computing devices were electronic glyphs, albeit from a limited range. Other media are newer arrivals on computer screens, especially when combined in integrated media. Media with associations with film-making and broadcasting have greater glamour than those associated with books and journals. • Perhaps issues concerning text display on computer screens seem to have been sufficiently well covered in the past, for example by Van Nes (1985), Verschelden and Konz (1988) or Rubenstein (1991). However, enough has surely changed to call for these older observations to be reviewed (Van Nes is about teletext, for example). The lack of attention to screen layout and typography in the HCI world is sadly exactly complemented by a lack of interest in HCI in the graphics community, including in design courses in education. Graphic design students in the new universities (usually former art schools) generally have a good education in the appreciation and use of type. However they often have little interest in computer screens, typically seeing computers as tools in the production of paper documents, rather than as media in their own right. In addition, their training often does not suit them to understand the value of ergonomics studies, or indeed of any kind of objective measures. There is mutual incomprehension between the ‘arts’ tradition of visual design education and the ‘science’ tradition of user-interface design. Visual designers are largely unfamiliar with the issues which we discuss below; do not recognise the potential of interactive text; or perhaps are simply repelled by the crudity of the tools, as they were by page-layout in the early days of word-processing. Perhaps there is a deprecation of language in general, arising from the fact that for most people its performance and comprehension are easy: ‘Ordinary speech, like colour vision or walking, is a paradigm of engineering excellence—a technology that works so well that the user takes its outcome for granted, unaware of the complicated machinery hidden behind the panels.’ Pinker 1994 Notoriously the AI community believed for some time that programming a computer to play chess would be more difficult than programming one to use natural language. This is an indicator of the lack of seriousness with which language in general has been approached in the computing community. Towards better design for text We need to deal briefly with the problematic term ‘Design’, since it is used to mean so many things. To some it means visual design alone, while to others structural aspects are paramount. These partial views arise partly from the divisions in education remarked on above. For multimedia, we argue, design should mean the integration of the visual, aural, interactional, conceptual, structural and technical aspects of the artefact. It is emphatically not surface decoration—perhaps more akin to architecture. We believe that the neglect of text as a visual component of the interface partly reflects a neglect of visual rhetoric in interface design generally, but this is too large a topic for this paper. We would only argue here that the aim in designing text for computer displays must be to assist in the process of communication. On this issue there are three basic positions. There is a view that text design for screens should be purely functional and confined to issues of legibility: Perfection is still defined in terms established by printers in the 15th and 16th centuries... [In an electronic space] the subtleties of type size and style may no longer be important to the writer’s or the reader’s vision of the text. Bolter 1991 The second view is that text should be designed with sensitivity to readership and the physical attributes of the substrate or display, but should seek to make the text ‘transparent’ to the content. Warde exemplifies this approach in ‘The crystal goblet: printing should be invisible’ (1955) in which she likens good typography to the design of a good wine-glass: ‘everything about it is calculated to reveal rather than to hide the beautiful thing which it was meant to contain.’ Tschichold (1958) also asks the designer to be a ‘loyal and tactful servant of the written word’. The third view, fundamental to the projects described below, is to ‘establish and amplify textual meaning’ (Poynor and Booth-Clibborn 1991). In this view, in addition to legibility there is what can be called the rhetoric of the design, the enriching contribution which design can make. For example, non-designers may be unaware that there is a huge literature simply devoted to choosing typefaces, on account of their ability to influence the reader’s overall perception of the content. The font, size, style, colour, hyphenation and spacing, and above all the relationship between the appearance and position of one part of the text and another, will all interact in enhancing or diminishing the communication in the text. There is also a role for text as a source of aesthetic pleasure in its own right. The projects we describe recognise that text can be inherently attractive. Writing on the use of lettering on buildings, Gray (1960) remarked: ‘There are qualities in virtue of which lettering may be architectural detail and ornament, quite apart from any message it may convey.’ Since the claims made for colour, video, sound and animation in multimedia are often based on the ‘appeal’ of these elements, it is odd that this aspect of text has been so neglected. Recently computer graphics animators have ‘rediscovered’ text as an object to be manipulated, animated and orchestrated as part of TV adverts, idents and stings. Electronic text technology It might seem that typography on computers is now approaching perfection; after all, most paper publications are now originated using computer typography. The range of fonts available is now vast. Widespread adoption of Postscript has largely solved problems of accuracy and resolution. Page layout software allows small and subtle adjustments. However, when type must be delivered on screen weaknesses start to emerge. In many cases, systemlevel software is capable of doing far more with text than is actually implemented in the multimedia authoring packages which run on those systems. Problems which persist include: resolution, spacing, orientation, anti-aliasing and font rendering. Resolution Spatial resolution in the display of desktop computers is improving, with 640x480 pixel displays regarded as standard, and 1024x768 displays becoming more common. Still this compares poorly with the resolution of a standard office laser-printer (600dpi) let alone a professional print publication (typically made from artwork rendered at 1200 or 2400 dpi). A double-page A4 publication provides 0.13 square metres of area. We are still a long way from approaching corresponding screen resolutions: our double-page would be 17.5 million pixels at 300dpi, compared with the 307 thousand pixels of our current 640x480 displays. Worse, it has been pointed out (Osborne 1994) that the authors of a paper like this one will typically surround themselves with open books as they write: under these circumstances, the paper documents are providing about 2.5 square metres of visible text-filled surface, compared with the 0.05 square metres of the screen. Spacing One of the first things that a typographic student learns is the importance of spacing. Judgments need to be made about the mean space between letters, words, lines and paragraphs and their permitted minima and maxima. These decisions are taken in the light of line-length, average paragraph length, type-size, font, hyphenation preferences, readership, likely viewing conditions and so on. The choices made affect readability (Van Nes 1985) as well as aesthetic pleasure. Page-layout tools such as Adobe PageMaker and Quark Express provide comprehensive spacing controls. They use kerning tables to optimally adjust the spacing between letter-pairs, and in addition permit hand-tuning of individual inter-character spaces. None of the multimedia authoring packages offers more than a fraction of these options. Orientation Designers sometimes gratuitously place lines of text upright when they would more usefully be presented in the normal way. However there are many occasions when alternative orientations are genuinely valuable. For example, column labels on tables are frequently set at an angle in order to save width without compromising readability. Cartographers always need some text items which are not horizontal. Multimedia packages do not support live text at any orientation but horizontal. Anti-aliasing Even though the font-design software for some platforms permits the definition of antialiased[1] fonts, and system software may also support anti-aliased typography, in practice most interactive media packages do not. This leads to the adoption of an awkward workaround. In order to get the superior appearance of anti-aliasing, text is prepared in a pixelediting package such as Adobe Photoshop, and imported into the multimedia package. Once there, it is just a pixel-map, having none of the advantages of live text, such as the ability to be sorted, searched or edited. To overcome these problems, some multimedia products have employed their own rendering of anti-aliased text (for example the Art Gallery and Wine Guide CD-ROMs created for Microsoft by Cognitive Applications), but built-in support for anti-aliasing in Interactive Media packages is long overdue.[2] Font rendering Until a few years ago, only those fonts which were represented by bitmaps of the correct size could be correctly imaged on desktop computers. With the launch of Adobe Type Manager it became possible for fonts to be rendered on demand on the basis of their Postscript geometry. Subsequently, TrueType was developed by Apple and Microsoft. Though Type Manager now supports both formats, there are odd weaknesses. For example, italic styles may be rendered poorly if the bitmap for that size is present, better if it is absent! Such a situation could not come into existence if typography for displays was taken seriously. Although ‘modern bit-mapped colour displays allow the designer to use the techniques of typography and graphic information design to express qualities beyond the simple content of the words’ (Crampton-Smith 1993) it is unfortunately the case that, from a designer’s point 1 Anti-aliasing is the technique which blends the edge pixels of graphic shapes with the pixels of the background in order to fool the eye into seeing smooth edges where in fact there is pixelation. 2 Macromedia Director offers anti-aliasing, but it is so slow and so poorly done that authors generally avoid it. of view, exciting developments like the World Wide Web represent a backward step. All the weaknesses of early word-processing are revived, including the lack of fully WYSIWYG tools, the absence of control over fonts, and poor controls over alignment and other aspects of layout. Hypertext: some design considerations The addition of interactivity to electronic text gives us hypertext. It also gives us the possibility of other kinds of interactive text, discussed below. The minimum requirement for a hypertext is that texts are presented such that, when the user interacts with certain parts, other texts (or in hypermedia, other media) are displayed. This functional definition leaves almost everything undecided in terms of how we might represent these elements and processes to the user. In what follows, we select a few examples to illustrate the range and variety of decisions which must be taken. Existing hypertext systems typically address only some of these issues. The text Although books contain semantic chunks—for example paragraphs—this chunking does not dictate how the text is paginated: text is interrupted when a page is full. This arbitrary cutting of the text has its electronic counterpart in the scrolling field. But we may prefer to break text logically, interrupting it where the sense dictates. Links Quite apart from the simple necessity to indicate to the user that certain text items are nodes connecting to others, there are many additional indications which might be made. To mention only a few, these might include: the type of link (for example, whether the incoming text will overlay or replace the current text; for hypermedia, the data-type of the linked material); whether the link has been recently used (Web browsers recently made this feature standard where it was previously almost unknown); the relationship between the current text and the new text. Transitions Transitional effects, connecting the current text to the incoming text, make strong use of spatial and cinematic metaphors. A few examples include: a wipe or scroll effect tends to suggest the arrival of a self-contained nugget of information; fades and dissolves suggest a certain distance between the two texts, analogous to the lapse of time indicated by similar effects in the cinema; barn-doors-open suggests an information space in which some information is under or within other information; focus-pull (used by one of our own projects described below) coaxes the user to give attention to particular parts of the text. Hypertext in a prototype CD-ROM In a project by Mark Osborne and Steven Scott[3], several improvements on the interface conventions of standard hypertext systems were developed. The project was a prototype for a populist educational CD-ROM about Quantum Physics. Some of the objectives of the proposed improvements were to: 3 The project was undertaken for the MA Design for Interactive Media in conjunction with System Simulation Ltd, a specialist software engineering and multimedia company. • encourage the user to read rather than skip the text • improve the user’s understanding of the context of each text item • disentangle the hypertext link indicators from the text itself • give a richer indication of what each link makes available Encouraging the user to read Advice on multimedia design repeatedly emphasises the need to minimise the number of words used (for example, Riley et al 1990). This does not take into account the user’s relationship to the text. Most computer users spend a considerable proportion of their time word-processing, viewing 200 or more words at a time out of documents comprising several hundred or even thousand words. The amount of text which people find acceptable is clearly not absolute, but is dependent on their needs at the time. Nevertheless, when text is presented to the user rather than being the user’s own creation, then large blocks of text can certainly be inimical to screen reading. Scrolling windows may well remain unscrolled, especially if the user does not know how much text, currently invisible, remains to be read. In order to overcome this problem, the prototype used no scrolling windows, instead organising all text in paragraphs or groups of paragraphs. There are other reasons to reject the scrolling windows of standard GUIs: they break a standard rule of information design, namely that the ‘furniture’ which supports the substantive information (in the case of scrolling windows, the text) should be visually subordinate (Tufte 1990). Instead, windows usually dominate the information which they should be supporting. However, scrolling windows do offer one advantage over discrete chunks of information: they provide a context for the current paragraphs by making it quick and easy to scroll to the preceding or following ones. Improving the user’s understanding of context In order to get the benefits of context without using scrolling windows, the project used ‘ghosts’—semi-transparent copies of the previous and next texts which were presented respectively to the left and right of, and partially overlapping, the current text. In this way users have the continuity of text associated with paper media, but their attention is focussed on a small part at any one time. Disentangling hypertext link indicators Hypertext systems generally differentiate those text elements which are active and lead to other data using typographic styles or standard annotation markers such as asterisks, daggers, and so forth. In either case, the readability of the text is affected. This is a problem of any annotated text, even on paper. In an introduction to the ‘Bible designed to be read as literature’, Sir Arthur Quiller-Couch remarks of traditional bibles: ‘Let us pepper the result over with italics and numerals, print it in double columns, with a marginal gutter on either side, each gutter pouring down an inky flow of references and cross-references. Then, and not till then, is the outward disguise complete.’ (Quiller-Couch 1950). The solution in the Quantum project was to disentangle most link information from the text, confining the indicator to a miniature icon at the periphery of the text column. Only in the case of single words requiring glosses was the convention of emboldening hot words used, with no italic or underlining. In order to tie text elements to their link icons, areas of tone extended under the phrases into the margin. The contrast of these areas to the main background was chosen to be the minimum necessary to distinguish them, again with a view to avoiding as far as possible any obstacles to smooth reading. A richer indication of what each link makes available Users usually activate a link in a hypertext in ignorance of what they will see next. They could use or reject links in an efficient and informed way, if only they had some idea what was to come. In the Quantum project, the mini-icons for hypermedia links always indicate the data-type of the material available. Users therefore know in advance if they will see a photograph, a video, a simulation or another text if they activate the link. This principle is extensible to other kinds of difference depending on the context: it could be used to differentiate between primary and secondary sources, mathematical and textual representations, or several other useful varieties of information. Text as interactive object The term ‘media integration’ has been used by some recently in place of ‘multimedia’. Text is usually the least integrated of all the media types, foreign to the multimedia interface, laid over it or beside it or segregated in a bordered window. In the Quantum prototype described, the hypertext had a more or less conventional aspect, albeit an improvement on other systems. However, attempts were also made to treat text as a truly integrated element of the user’s overall interactive experience. The objective was that the text should become seamlessly interconnected. Text focus-pull In one section of the prototype, about 200 words of text were presented on screen at once. In order to actively engage users and assist their concentration, the text was divided into blocks each comprising one or two paragraphs. On first appearing, all paragraphs were out of focus: only when the user pointed at them did they pull into focus. The focus/defocus effect for the prototype was a simple piece of graphic processing done using Adobe Photoshop. The user had to engage with the project in order to read it. An additional advantage of the technique was that it served as a subtle reminder to the user of the active role of the observer so fundamental to quantum physics. Thus even the visual form of the text was playing its full part in projecting the content of the project. A bicycle in the works Other attempts were made to intrigue and engage the user, but not at the cost of distracting the user from the content. At one point in the text provided by the author, bicycles are mentioned as part of a metaphor of quantum effects. It would clearly have been excessive and distracting to present the user at this point with a picture of a bicycle! Instead, a faint shadow of a cyclist flits across the text that the user is reading. The text is slightly distorted as the cyclist passes, restored to normality on his departure. The medium of the text—as well as the text itself—is helping to convey the message. For other aspects of the project see Osborne (1994). Graphics-sensitive text We hope we have begun to illustrate the idea of text as an equal partner with other media in the user’s experience of the multimedia product. On this theme another of our projects has also sought ways of fully integrating text with graphics, video and animation. In a prototype undertaken for the Technology Enhancement Programme of the Engineering Council by Alison Murray, Stacey Pogoda-Curtis and Joolz Pohl, an interesting problem had to be solved: users of the proposed interactive educational product were to be presented with a rich variety of information in various media about an everyday engineered object, such as a handheld power-drill. They were also to have access to a library of terms and concepts. The intimate mix of text and graphics developed for the project allowed combinations of visual material and supporting text to appear on the screen together in a truly integrated way. For example, when a set of terms are summoned to the screen by the user and subsequently dismissed, the user can choose to keep, in their relevant positions on the photographic illustration, those terms which are pertinent to the current object. In this way users are free to browse any parts of the library, but are subtly guided into concentrating on the most relevant terms. Text and image are interdependent and either can be interrogated in similar ways—pointing, clicking, dragging—so obviating the need for the usual differing interface devices for interacting with text and interacting with graphics. Text as graphicsFinally, the project used lettering as both message and decoration; in other words, it was usedto provide function and pleasure for the user, as advocated by Gray (above). Far from thevisual form of text being subordinated to other elements, some text items were used on alarge scale as interface elements in their own right (it is said that William Morris decided toadd typography to his other activities on attending a lantern-slide show of hugely enlargedletter-forms). We would like to think that bringing text into the centre-stage of multimedia,may help to refocus attention on the beauty of the text and the power of words. Related issuesare discussed in Pogoda-Curtis (1995) ReferencesBolter, Jay David Writing space—the computer, hypertext and the history of writing 1991Lawrence ErlbaumCrampton-Smith, Gillian ‘The Art of Interaction’ In Doors of Perception 1993 AmsterdamGray, Nicolette Lettering on buildings 1960 Architectural Press, London and Rheinhold, NewYorkHanneman, Jörg and Thüring, Manfred ‘What matters in developing interfaces forhyperdocument presentation?’ in Schuler, W; Hanneman, J and Streitz, N (eds.) Designinguser interfaces for hypermedia 1995 Springer, BerlinHardman, Lynda ‘Experiences in authoring hypermedia: creating better presentations’ inSchuler, W; Hanneman, J and Streitz, N (eds.) Designing user interfaces for hypermedia1995 Springer, BerlinHarrison, Susan M ‘A comparison of still, animated, or non-illustrated on-line help withwritten or spoken instructions in a graphical user interface’ SIGCHI ’95 (Denver, ColoradoMay 7-11 1995) In Human Factors in Computing System proceedings, annual conferenceseries, 1995 ACM SIGCHI, pp82-89Henno, Jaak ‘Some guidelines to PC hypertext authoring’ Hypermedia in Vaasa ’93 (Vaasa May 24-26 1993) In Hypermedia in Vaasa proceedings, Vaasa Institute of Technology, 1993pp121-125INTERCHI ’93 (Amsterdam April 24-29 1993) Human Factors in Computing Systemproceedings, 1993 ACM New YorkLaurel, Brenda; Oren, Tim and Don, Abbe ‘Issues in multimedia interface design: mediaintegration and interface agents’ in Blattner, Meera M and Dannenberg Roger B (eds)Multimedia interface design, ACM Press, New York 1992, pp53-64Loo, Joel Peing-Ling ‘A knowledge-based approach to hypermedia’ in Waterworth, John(ed.) Multimedia technology and applications 1991 Ellis HorwoodOgozalek, Virginia Z ‘A comparison of the use of text and multimedia interfaces to provideinformation to the elderly’ In SIGCHI ’94 (Boston, Mass April 24-28 1994) in HumanFactors in Computing System proceedings, annual conference series, 1994 ACM SIGCHI,pp65-71Osborne, Mark Issues of form and content in electronic text (unpublished MA dissertationMiddlesex University 1994)Pogoda-Curtis, Stacey Interacting with information: designing an invisible interface for theTEP prototype (unpublished MA dissertation Middlesex University 1995)Pinker, Steven The language instinct: the new science of language and mind 1994 Penguin,UKPoynor, Rick and Booth-Clibborn (eds.) Typography now—the new wave 1991 InternosQuiller-Couch, Arthur, quoted in The Bible designed to be read as literature ed. ErnestSutherland Bates Heinemann London undated (20th century, 1950s?)Riley, David (with Richard Millwood, Alan Edis, Mike Figg and David Riddle) Design foractive learning with HyperCard 1990 King’s College LondonRubenstein, R ‘Digital typography: improving the quality of type on screen and paper’Tutorial presented at CHI'91. (Tutorial no. 15) In Reaching through technology CHI'91.Proceedings of the conference on human factors in computing systems, 27 Apr 2 May 1991,New Orleans, Louisiana, USASIGCHI ’94 (Boston, Mass April 24-28 1994) Human Factors in Computing Systemproceedings, annual conference series, 1994 ACM SIGCHISIGCHI ’95 (Denver, Colorado May 7-11 1995) In Human Factors in Computing Systemproceedings, annual conference series, 1995 ACM SIGCHITschichold, Jan ‘On typography’ (1952) reproduced in Tschichold, Jan The formof thebook—essays on the morality of good design, Lund Humphries 1975Tufte, Edward Envisioning information 1990 Graphics Press, Cheshire, ConnecticutVan Nes, FL ‘On the use of layout, colour and typography on visual display terminals’ in IPOannual progress report 1985. 1985 (IPO report, no. 20) Institute for Perception Research.Verschelden, M and Konz, S ‘Absolute and relative ratings of type fonts and styles’ inMegaw, ED (ed.) Contemporary ergonomics, 1988: proceedings of the Ergonomics society's1988 annual conference Manchester, 11 15 April, 1988 Taylor & Francis, LondonWarde, Beatrice The Crystal Goblet 1955 Sylvan Press, London